refactor(docker): split gateway/supervisor Dockerfiles and use native rust builds by TaylorMutch · Pull Request #1316 · NVIDIA/OpenShell

TaylorMutch · 2026-05-12T01:19:51Z

Summary

Splits the combined Dockerfile.images into per-image Dockerfiles (gateway and supervisor) and switches local image builds to use native Rust binaries instead of compiling inside Docker. This matches the CI pipeline and makes local image rebuilds much faster because the host's cargo target cache and sccache are reused across iterations.

Related Issue

N/A — incremental cleanup of the Docker build pipeline.

Changes

deploy/docker/Dockerfile.gateway (new) — nvcr.io/nvidia/distroless/cc:v4.0.4 base, GNU-linked binary at /usr/local/bin/openshell-gateway. Image dropped from 194MB → 101MB.
deploy/docker/Dockerfile.supervisor (new) — scratch base, static musl binary at /openshell-sandbox. Static linkage lets the image stay scratch while remaining executable in the Kubernetes init-container copy-self path. Image is 21.7MB.
deploy/docker/Dockerfile.images (deleted) — replaced by the split files.
deploy/helm/openshell/skaffold.yaml — each artifact now invokes tasks/scripts/docker-build-image.sh, which stages binaries natively (cargo / cargo-zigbuild) before running the Docker build. Drops BUILD_FROM_SOURCE=1 and the cross-build.sh dependency from the supervisor build.
tasks/scripts/stage-prebuilt-binaries.sh — supervisor now builds against musl (x86_64-unknown-linux-musl / aarch64-unknown-linux-musl); gateway stays on GNU.
tasks/scripts/docker-build-image.sh — routes gateway / supervisor targets to the new per-image Dockerfiles.
.github/workflows/rust-native-build.yml — supervisor binary build uses zig cc musl wrappers so the static binary can be cross-compiled in CI.
crates/openshell-sandbox/src/sandbox/linux/seccomp.rs — adds a local SYS_kexec_file_load constant for musl/aarch64. libc 0.2.185 omits this symbol from its musl/aarch64 bindings, preventing the supervisor's seccomp filter from compiling for that target. The constant matches the kernel's syscall number (294) so the filter still blocks fileless kernel-image loads.
architecture/build.md — rewrites the Container Builds section around the native-first pipeline and the per-image runtime choices.
Minor doc touchups in crates/openshell-driver-podman/README.md and .agents/skills/debug-openshell-cluster/SKILL.md pointing at the new Dockerfile paths.

Testing

Local native build of both binaries succeeds via tasks/scripts/stage-prebuilt-binaries.sh all (gateway GNU, supervisor static musl).
tasks/scripts/docker-build-image.sh gateway and … supervisor produce running images; --version smoke test passes for both.
mise run helm:k3s:create + mise run helm:skaffold:run deploys cleanly into a local k3d cluster; gateway pod goes 1/1 Running with healthy startup logs.
Sandbox creation via the CLI exercises the Kubernetes init-container copy-self path: the supervisor scratch image copies the static musl binary into the agent pod's emptyDir, and the agent container runs it inside a non-glibc image (busybox) successfully.
mise run pre-commit clean for files changed in this PR. (Unrelated cargo fmt drift in driver.rs will be handled in a separate PR.)
CI image-build workflow needs to validate the new musl supervisor build for both architectures.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated

… images Drop the in-Docker BUILD_FROM_SOURCE path so both images consume only prebuilt binaries staged natively via tasks/scripts/stage-prebuilt-binaries.sh. This mirrors what CI does and reuses the host's cargo target cache and sccache across rebuilds. - Dockerfile.gateway: nvcr.io/nvidia/distroless/cc:v4.0.4 base (the 4.0.0 tag does not exist on nvcr.io; the registry uses a v prefix). GNU-linked binary copied to /usr/local/bin. - Dockerfile.supervisor: scratch base, static musl binary. Static linkage lets the image stay scratch while still being executable as a Kubernetes init container. - skaffold.yaml: each artifact invokes tasks/scripts/docker-build-image.sh, which stages the binary natively (cargo / cargo-zigbuild) and then builds the image. Drops the cross-build.sh dependency from the supervisor build. - seccomp.rs: add a local SYS_kexec_file_load constant for musl/aarch64. libc 0.2.185 omits the symbol from its musl/aarch64 bindings, so the supervisor's seccomp filter previously failed to compile for that target. - architecture/build.md: describe the native-first pipeline and per-image runtime choices. Local validation: gateway image 101MB (was 194MB), supervisor image 21.7MB. helm:skaffold:run deploys cleanly; the static musl supervisor binary runs correctly in a non-glibc agent container.

copy-pr-bot · 2026-05-12T01:19:55Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-05-12T01:29:07Z

Label test:e2e applied, but pull-request/1316 is at {"messa while the PR head is f763611. A maintainer needs to comment /ok to test f76361152f81ad7c235d3518f83eba8b8397aec1 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

TaylorMutch · 2026-05-12T01:30:20Z

/ok to test f763611

Fixes formatting drift flagged by `cargo fmt --all -- --check`.

TaylorMutch · 2026-05-12T01:38:45Z

/ok to test 59a089afd5f44a4c46cf5a4eb3712f99066014fc

copy-pr-bot · 2026-05-12T01:38:49Z

/ok to test 59a089afd5f44a4c46cf5a4eb3712f99066014fc

@TaylorMutch, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

TaylorMutch · 2026-05-12T01:39:27Z

/ok to test 59a089a

TaylorMutch · 2026-05-12T01:55:41Z

+// libc 0.2.185 omits `SYS_kexec_file_load` from the musl/aarch64 bindings even
+// though the kernel exposes syscall 294. Fall back to the literal so the
+// supervisor's seccomp filter still blocks fileless kernel-image loads when
+// built statically against musl on aarch64.
+#[cfg(all(target_arch = "aarch64", target_env = "musl"))]
+#[allow(non_upper_case_globals)]
+const SYS_kexec_file_load: libc::c_long = 294;
+#[cfg(not(all(target_arch = "aarch64", target_env = "musl")))]
+use libc::SYS_kexec_file_load;


This failed to work on macOS locally, so was added to get it working with a static build.

github-actions · 2026-05-12T02:11:31Z

Label test:e2e-kubernetes applied for 59a089a. Open the existing run and click Re-run all jobs to execute with the label set. The E2E Gate check on this PR will flip green automatically once the run finishes.

elezar · 2026-05-12T06:18:06Z

        libc::SYS_finit_module,
        libc::SYS_delete_module,
        libc::SYS_kexec_load,
-        libc::SYS_kexec_file_load,


Should this be dependant on the constant defined or imported above? https://github.com/NVIDIA/OpenShell/pull/1316/changes#r3223215941

elezar · 2026-05-12T06:22:24Z

+# the dynamic loader needed by the GNU-linked gateway binary while keeping the
+# attack surface small.
+
+ARG GATEWAY_BASE_IMAGE=nvcr.io/nvidia/distroless/cc:v4.0.4


We would need to update

OpenShell/.github/workflows/e2e-gpu-test.yaml

Line 61 in 9ea94b6

GPU_PROBE_IMAGE="$(awk '$1 == "FROM" && $3 == "AS" && $4 == "gateway" { print $2; exit }' deploy/docker/Dockerfile.images)"

for this change.

The image choice in the job is pretty arbitrary and should work as long as nvidia-smi can run there. I wanted to avoid using ubuntu directly, but that would also be possible.

elezar · 2026-05-12T06:24:52Z

    DOCKER_TARGET="supervisor"
+    DOCKERFILE="deploy/docker/Dockerfile.supervisor"
    ;;
  supervisor-output)


out of scope for this PR, but would matching on supervisor|supervisor-output make sense here?

drew and others added 2 commits May 11, 2026 15:40

wip

2387d04

TaylorMutch marked this pull request as ready for review May 12, 2026 01:23

TaylorMutch requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners May 12, 2026 01:23

TaylorMutch added the test:e2e Requires end-to-end coverage label May 12, 2026

style(driver-kubernetes): apply cargo fmt to driver.rs

59a089a

Fixes formatting drift flagged by `cargo fmt --all -- --check`.

TaylorMutch commented May 12, 2026

View reviewed changes

TaylorMutch added the test:e2e-kubernetes Requires Kubernetes end-to-end coverage label May 12, 2026

elezar reviewed May 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(docker): split gateway/supervisor Dockerfiles and use native rust builds#1316

refactor(docker): split gateway/supervisor Dockerfiles and use native rust builds#1316
TaylorMutch wants to merge 3 commits into
mainfrom
tmutch/fix-image-builds

TaylorMutch commented May 12, 2026

Uh oh!

copy-pr-bot Bot commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

TaylorMutch commented May 12, 2026

Uh oh!

TaylorMutch commented May 12, 2026

Uh oh!

copy-pr-bot Bot commented May 12, 2026

Uh oh!

TaylorMutch commented May 12, 2026

Uh oh!

TaylorMutch May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

elezar May 12, 2026

Uh oh!

elezar May 12, 2026

Uh oh!

elezar May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

TaylorMutch commented May 12, 2026

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

TaylorMutch commented May 12, 2026

Uh oh!

TaylorMutch commented May 12, 2026

Uh oh!

copy-pr-bot Bot commented May 12, 2026

Uh oh!

TaylorMutch commented May 12, 2026

Uh oh!

TaylorMutch May 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

elezar May 12, 2026

Choose a reason for hiding this comment

Uh oh!

elezar May 12, 2026

Choose a reason for hiding this comment

Uh oh!

elezar May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants